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ABSTRACT 



A test based on the Operations Analysis Curriculum 
at the United States Naval Postgraduate School was 
administered to 102* Naval Officers. All examinees were 
graduates or students of the Operations Analysis Curricu- 
lum and/or officers holding Operations Anslysts billets in 
the Navy. The sub-sample, 34 examinees, consisting of 
officers holding Operations Analysts billets and/or Operations 
Analysis graduates was not sufficient to make adequate 
statistical determination of the measure of effectiveness 
proposed in a suggested methodology. The data gathered 
did crudely support hypothesized learning and forgetting 
curves and suggested that the effectiveness of Operations 
Analysis graduates assigned directly to Operations Analysts 
billets immediately after graduation is much enhanced compared 
to graduates who are returned first to fleet operational 
billets. The effectiveness of Operations Analysis trained 
officers in Operational Analyst billets was shown to be 
quantitatively and subjectively significantly superior to those 
with no formal Operations Analysis training. These results 
indicate that Naval assignment policies should be reviewed 
in hopes of assigning more Operations Analysis trained 
officers (consistent with other requirements) to these 
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billets. Further investigation of the results of the test 
vehicle and other statistics common to Operations Analysis 
graduates yielded a feasible procedure with which to 
augment the screening of prospective Operations Analysis 
students. Final Quality Point Rating, an acceptable 
measure of performance, had a . 611 + correlation with four 
readily available statistics. 

This evaluation suggests that further study in this 
area has great promise in yielding useful measures of 
effectiveness for all personnel filling billets requiring post- 
graduate education, provided a more effective method is 
employed to insure completion of the required test 
instrument( s ) . 
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CHAPTER I 



I INTRODUCTION 



In May of 19 62 + Commander P. D. Roman, and 
Lieutenant Commanders K. D. Russell and J» M. Dunlop 
posed a highly interesting and promising methodology designed 
to evaluate the effectiveness of the utilization of technically 



involved in this suggested Methodology were Naval Officers 
who have received postgraduate education. The main 
motivation for this proposed model was the fact that no 
previous quantitative study had been made to determine if a 
better balance of career duty assignments should be devised 
which would allow officers to attain the operational knowledge 
and experience requisite to Military Command and simultane- 
ously approach maximum effective use of their technical skills 
(sub-specialties). Because the acquisition of technical skills 
is costly , as well as necessary, finding an optimal procedure 
to utilize these skills, consistent with other requisites, is 
virtually a "must." The pure rationality of cost-effectiveness 
alone supports this supposition. 

Commander Roman, et, al., constructed a test instru- 
ment modeled on the Operations Analysis curriculum at USNPGS 



trained personnel within 




The personnel 



1 



which would, hopefully, measure the native ability and 
technical knowledge of Naval Officers who are holding 
Operations Analysis billets and/or graduates of the Oper- 
ations Analysis Curriculum. It was their hypothesis that 
by categorizing these officers into several distinct groups 
the statistical results of their performance on this test 
would lead to a more efficient system of career planning 
for these officers. If this procedure proved successful 
for the Operations Analysis sub-specialty, it would obviously 
have applications for all officers who are technically trained. 
The authors of the referenced methodology have presented 
an admirable treatise on the need for the execution of such 
a quantitative study. 

In September of 1964 the authors of this paper mimeo- 
graphed the test instrument and distributed it to officers 
holding Operations Analysis "P" coded billets and/or graduates 
of the Operations Analysis Curriculum at USNPGS. A 
total of 34 returns from, a population of 197 officers was 
realized. In January of 1965 the test was also administered 
to 36 second year Operations Analysis students at USNPGS 
scheduled to graduate in May 1965 . In November of 1964, 

34 first year Operations Analysis students completed the 
aptitude portion of the test vehicle. 
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The purpose of this paper is three-folds 

I. To determine the adequacy of the test as a 
measuring instrument, i.e., does it measure what it is 
supposed to measure? 

II. To carry out, where possible, the statistical 
procedures suggested by Commander Roman, et. al. , and 
analyse their usefulness and/or implications. 

III. To analyse the results of a multiple linear 
regression analysis performed on the CDC 1604 Computer 
to develop a statistical means to augment selection 
procedures for input to the Operations Anslysis Curriculum. 

The following chapter details the analysis of the data 
for each of these purposes. 
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CHAPTER II 



THE ANALYSIS 

I. Determination of the adequacy of the test instrument. 

The test instrument used in this study consists of a 
background questionaire and three test parts. The question- 
aire is designed to obtain the examinee's educational background, 
a history of his duty assignments, a listing of graduate and 
undergraduate courses completed and the examinees opinions' 
as to what courses he is lacking that are required in his 
billet. The main purpose of this part of the test is to 
stratify the examinee according to educational background. 

The other information gained will provide further refinement 
of this stratification. The adequacy of this questionaire is 
purely subjective. The authors feel that the information 
provided by the questionaire will allow assignment of testees 
to logical categories of interest to this study. 

Part I of this test is an aptitude test designed to 
provide a measure of the examinee's ability in problem solving, 
the only major factor accepted as effecting current measures 
of intelligence. ^ This test was constructed, from investigations 

"^Editorial. Federal Education, You're in the Classroom 
Now. Time, 83, 3» January 17, 1961+ s 72 



k 



of aptitudes of high level personnel, under the direction of 



Dr. <J. P. Guilford at the University of Southern California, 



results of this exhaustive and authoritative study has led to 
the formulation of an excellent test to measure native ability. 

It follows that the results of this test will allow examinees 
to be rated according to inherent abilities. This stratification, 
coupled with background areas, will provide an excellent means 
for comparison of technical skills in the selected categories. 

Part II of the test deals strictly with the retention 
of fundamental concepts of the basic courses within the 
Operations Analysis Curriculum: namely, Advanced Calculus, 

Linear Algebra and Probability theory. This test was 
formulated from suggestions of Professors of the Mathematics, 
Physics and Operations Analysis Departments of USNPGS. 

It is indeed a moot question as to whether or not a good 
working knowledge of these courses measures, with any degree 
of accuracy, the ability or success of an individual as an 
Operations Analyst. There can be little doubt, however, 
that proficiency in these fields does reflect some measure of 
the examinees technical abilities. Beacuse we are once again 
caught in a subjective (or qualitative) area, at this point we 
will assume that the combined opinions of these recognized 




The authors will assume that the 
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educators provides a good cross section of the technical 



knowledge required of an effective Operations Analyst, and 
the results of Part II of this test will yield, at least, a 
relative measure among the groups of officers of their 
abilities as Operations Analysts. Relative performance is, 
after all, a very important aspect in our real world and 
this relative performance is in essence one major objective 
of this study. 

Part III is a practical test designed to evaluate the 
examinees' ability to recognize the applicability of a class of 
methods to specific problems. The situations presented in 
these problems do not have clean-cut answers, but are 
designed to determine how familiar an examinee is with an 
"accepted or proven" Methodology as related to well-known 
Operations Analysis problems. The answers to the situations 
posed are nothing more than a mean combination of opinions 
expressed by noted analysts and professors at USNPGS. 
Once again we are using the general reasoning of the 
preceding paragraph in stating that this portion of the test 
will display a relative measure of how familiar an examinee 
is with the "accepted" tactics of Operations Analysis. 

Thus far we have been concerned with the question of 
whether the test will measure what we want it to measure. 
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Is it a valid indication of the technical abilities we are 



trying to measure? To this point we have tried to answer 
these questions in the affirmative by the use of subjective 
reasoning. This procedure is necessary because the deter- 
mination of test validity does not readily or easily lend 

2 

itself to meaningful quantitative analysis. Despite this 

* 

dilemma, certain mathematical techniques do allow a degree 
of quantitative determination of adequacy to be calculated. 
The entire question of determining the adequacy of this test 
could be measured by two broad criteria; VALIDITY and 
RELIABILITY. Reliability is defined as a measure of how 
faithfully the test allows the examinee to display the true 
percentage of the questions presented in the test to which 
he actually knows the answers. A mathematical presentation 
of this reliability follows. 

Doctor <J. P 0 Guilford has developed a mathematical 

model to evaluate the reliability of any test. Because the 

entire development is extremely lengthy we shall present 

only the basic assumptions and final forms of the reliability 

3 

equations. 

2 

Guilford, <J. P» Psychometric Methods, McGraw Hill 
Co., New York, 1954? 36 

3 Ibid: 344-409 
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The effects of these variations have been reduced as 



much as possible by the use of an a priori weighting of raw 
total scores. 

U) s - R t . £=^ . - w , 



where: S = Adjusted Score, 

R = Number of Right Answers, 

W = Number of Wrong Answers, 

K = Number of Alternate Responses to each 
item. 

This method of scoring is designed to nullify the small but 
not necessarily minute possibility that the examinee may 
guess the correct answer. 

Doctor Guilford has postulated that, in theory, a 
regression equation could be calculated between observed 
scores and the true score (exactly how many questions the 
student knows the answers to) as depicted in Figure 1. 

In the development of this theory Dr. Guilford postulated 
that in any academic area a test of infinite length would be 
necessary to completely describe an examinee's knowledge of 
the subject. In like manner the authors have denoted true 
score by the symbol 00 • 




Regression line. 

Standard deviation of the 
error in the measurement. 



FIGURE 1 
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The following assumptions were used in the mathematical 
development : 

I Mg = Mean error = 0. 

II = Correlation of true and error scores = 0 . 

III ^,£>2. - Error score correlation in any two forms 

of the same test = 0. 

IV Distribution of errors is normal. 

It then follows that: 

Mqq = M^. + Mg = M^. 



and 



M oo = ^7 + where \J £ = variance of L 0 

This leads to the logical definition of reliability of a test 
as the proportion of true variance in obtained test scores 



or 



( 2 ) 
where : 



& = 



Nl c© 

Tv 



Vr.-t = Test reliability, 

V"c» =• True variance of test scores 

= Observed variance of test scores. 

% 

Because of the difficulty in obtaining Sj~0O equation (2) has 

been reduced (closely approximated) as follows: 

& 

( 3 ) 

where: 



K* - 1 - Ts j 

J 

Q = Difference between the scores on even 






items and odd items on the test, 

= Variance of these differences, 

= Variance of obtained total scores. 

The variables in equation (3) can be readily calculated for 

any test. 
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A final consideration must be made in view of the 



fact that the test in question was timed and therefore 
speeded to some extent. Speed does detract from 
performance which we do not want reflected in reliability. 
The final form of this reliability equation is 

(4) Im = Yt* “ ) 

N t e 

where: irf\ = Reliability of slightly speeded test, 

Kt = as before, 

U_ = mean number of unattempted items, 
rr-^ 

= variance of total test error scores. 
Equation (4) is a close approximation to true reliability, as 
previously defined, of a speeded test provided that: 

jl £0.3 



'te 



where : Tu. = Standard deviation 

of unattempted test 
items, 



and 



- '' 3 . 

N W 






vv 



= Standard deviation of 
test items answered 
incorrectly. 



As suggested by Commander Roman, et. al., examinees 
were placed in the following categories: 

(1) USNPGS Students nearing completion of their 
final year of the curriculum. 

(1A) USNPGS Students in second term of first year 
of Operations Analysis Curriculum. 
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(2A) Graduates who have been assigned (and are in) 
directly to Operations Analysis billets. 

(2B) Graduates who have never been associated with 
Operations Analysis billets. 

(2C) Graduates who have completed a direct assignment 
in Operations Analysis billets and are now in unassociated 
activities . 

(2D) Graduates who were not immediately assigned 
Operations Analyst billets but are now serving in that 
capacity. 

(3) Non-Graduates who are presently serving in 
Operations Analyst billets. 

Clearly all examinees involved in this study fall into one 
and only one of these categories. The chosen method of 
attack is to compare various categories by means of forgetting, 
learning and re-learning curves in hopes of deducing a useful 
measure of effectiveness. As a common basic for comparison, 
all examinees were stratified by comparing their results of 
Part I (Inherent Abilities) with the mean score of Part I 

f 

for Category J.« This procedure is logical because present 
students should be more familiar with the basic fundamentals 
of the courses test Parts II and III are concerned with. 
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To be specific, a ratio of an individual’s score on Part I 
to the mean score of Category I on Part I was multiplied 
by the scores on Part II and III and this figure compared 
to the same mean score attained on Parts II and III by 
Category I . 

From equations (3) and (i+) and the information displayed 
in Appendix II the reliability of this entire test was calcul- 
ated to be = .90224.. Category I students were chosen as 
a reliability base for the following reasons; 

1. All data and comparisons are to be based on their 
performance. 

2. Strict time limitations were imposed on these 

testees and we feel that their test conditions are more in 

keeping with the stipulations and assumptions made in the 

2 + 

development of the reliability equations. 

In the final analysis, validity has to do with what test 
scores measure and what they will predict. A score is valid 
for predicting anything with which it correlates, where 

’’any thing” does not include the score itself, for a self 

5 

prediction has to do with reliability. We chose to predict 

^"Guilford, Co P. Psychometric Methods, McGraw Hill 
Co., Inc., New York, 19524-: 366 

5 Ibid: 398 
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final graduate level QUALITY POINT RATING* of 
Operations Analysis graduates of USNPGS. Here we 
have assumed that the degree of successful completion of 
the curriculum (hence a measure of technical ability) is 
measured by the reliability of the criterion of Q.P.R.; 
although the reliability of Q.P.R. as a criterion is not 
known, it is the only quantitative measure we have for 
comparison and have therefore assumed, for the purposed 
of this study, it to be 1.0. The results of a multiple 
linear regression analysis (Appendix IV, pp„50 ) shows that 
the correlation coefficient of Q.P.R. to various parts of 
the test scores is a minimum of .5978 and a maximum of 
.6142. As shown in Appendix IV, it was found that Part 
II versus Time in Operations Analyst billet or Time since 
Graduation, Part I and Category type had a multiple 
correlation of .8210, indicating that technical knowledge is 
extremely dependent upon inherent ability and the way time 
is utilized after aquisition of these abilities. Since the 
reliability of this test is a high .9024 and its overall 
correlation to a real life criterion is high, it follows that 
the quantitative adequacy of this test is very high. 

* Hereafter referred to as Q.P.R. , with a maximum of 

3-0. 
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Within the assumptions stated, the quantitative 
reliability and validity calculation and the previous subjective 
verifications, the overall adequacy of this test is excellent. 

One final word on this "sticky" subject. This test was 
sent to Mr. R.P» Richardson, Head Operations Analyst for 
LI NG-TEMP CO-VAUGHT ASTRONAUTICS in Dallas, Texas 
for his comment as well as the reaction of his fellow employees. 
Mr. Richardson replied that in his opinion, and the opinion of 
his collegues, the overall design of the test should yield a 
practical relative measure of the technical tools of Operations 
Analysts . 

II, Comments on the suggested method and proposed measure 
of ef f ecticeness . 

In the suggested methodology, two families of curves were 
to be plotted and studied in hopes of arriving at a useful meas- 
ure of effectiveness. The first family of curves was to be 
a plot of weighted adjusted score (as described in I above) for 
categories 2A, 2B , 2C versus time. The courses were post- 
ulated to appear as in Fisure 2. 




<t iyj 



Q_ = level of knowledge for 
. category 1 students. 



Time, in months, since graduation. 



FIGURE 2 
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The second family of courses were to be concerned with 
the process of learning and relearning basic technological 
knowledge and were hypothesized to appear as in Figure 3 . 




The plots of Figure 2, if correct, would indicate that 
intervening duty assignments between acquisition of technical 
knowledge and application in the given field results in a 
larger loss of the tools required. A useful M.O.E. was to 
be constructed from the type graphs of figure 3. For 
example: if "b" in figure 3 indicates the ordinal value of 

the asymptote of curve E, the ratio of curriculum time to 
curriculum time plus time required for category 2D (graduates 
who are essentially re— learning the technical tools) would 
yield a value between 0 and 1 , a higher value correlating to 
a more effective utilization of the technical resorurce and 
would imply a certain percentage of effectiveness. If this 
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ratio were to be drastically low it would demonstrate the 
need for assigning the technically trained to associated 
billets immediately after graduation to more effectively 
utilize the costly process of educating these individuals. 

Appendix III contains plots of weighted adjusted 
scores on Part II versus Time for the above mentioned 
categories. These graphs have the approximate shapes of 
Figures 2 and 3« At this point this particular plan of 
attack breaks down. As previously stated, 34 returns 
out of a population of 197 was realized. This resulted in 
a maximum of 14 points for the learning curve for category 
3 and a minimum of two points for category 2C. It is a 
well known fact that little credence can be placed in a 
statistical procedure unless: 

I. The sample size is very large (.9 to .95) in proportion 

£ 

to the population size, or 

II. One can safely assume knowledge of the exact distri- 
bution of the random variables observed, and/or have control 

n 

over the method of selecting the sample size. 

£ 

Burington, R. S c , and May, D. C„ Handbook of 
Probability and Statistics with Tables, Handbook Publishers, 
Inc., Sandusky, Ohio 1953s 170 

7 Ibid: 170-178 
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Clearly none of these proposed curves contain data 
satisfying any of these conditions. At most, one can say 
that the data so far gathered crudely support the hypothe- 
sized curves and that future study along these lines has 
more than a possibility of being fruitful. 

If, however, we consider the sample size to consist 
of only two groups, Operations Analysis graduates and Non- 
Operations Analysis graduates, several meaningful implications 
can be garnered from the test results. (Group I Operations 
Analysis Graduates, Group II Non-Operations Analysis 
Graduates) . The mean score on Part I for Group II was 
6.0 points below Group I indicating a significantly lower 
proficiency in the area of problem solving. The mean scores 
on Parts II and III were respectively , 3 ahd'4 times as high 
for Group I as for Group II. These mean scores had a 
standard deviation of less than 7.0 for the "worst" case. 
Group I's comments on the courses that they thought they 
were lacking were of a highly specialized nature (Dynamic 
Programming, Specialized Methods of Cost-Effectiveness, 
etc.) while Group II's overall response indicated they were 
lacking in the most rudimentary areas such as basic prob- 
ability and statistics, linear programming, calculus, etc. 

Only two officers in Group II had any postgraduate education. 
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These disparities in relative performance of Group II to 
Group I exist despite the fact that the mean time in 
billet for Group II is 19 months - only two months less 
than time required for completion of the Operations 
Analysis Curriculum, thus indicating that on-the-job 
training is not very effective,. From these facts there 
exists a clear implication that there is a significant 
difference in the technical abilities and therefore effective- 
ness of Group I and Group II as practicing Operations 
Analysts . The least one could say is that Group I displays 
a marked advantage over Group II in the technical tools 
needed in the field of Operations Analysis . 

One might say this is not surprising - any logical 
person would safely assume an Operations Analysis trained 
person to be more effective than one not so trained. This 
conjecture is most likely true in highly specialized fields 
(such as microelectronics), but cannot be safely assumed 
for fields such as Operations Analysis. Operations Analysis 
is concerned with optimal solutions to real life problems 
arrived at by the applications of all sciences and the rational 
logic intrinsic to them. Therefore it could just as well be 
said that an officer with a good working knowledge of "the 
sciences" has an excellent chance of being an effective 
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Operations Analyst. Although all examinees in Group II 
have Bachelor's degrees in fields requiring the basic math- 
ematical and scientific technologies used by the Operations 
Analyst, the quantitative results of this study indicate 
that their ability to effectively apply these tools to 
Operations Analysis problems is relatively low. 

Why is all this important? The Navy has stated that 
its policy in the future will be to continue to educate 

officers at the Postgraduate level in ever-increasing 
8 

numbers. Whatever their reasons may be for arriving at 

this decision, the cost of training these officers has or 

will be expended and cannot be recovered. It therefore 

logically follows that the most effective use of these 

officers will yield the maximum return for the money spent. 

In the past, many Operations Analysis trained officers have 

never held Operations Analyst billets and many officers 

have been assigned to these billets so long after their 

training as to nullify the maximum effectiveness they could 

9 

display. It is highly likely that this situation exists in other 
technical fields within the Navy. 

g 

OPNAV Instruction 1040*2 dated 9 December 1963s 2 

g 

Roman, P. D., Russel, K. B., and Dunlop, «J. M. 

A Suggested Method for Measuring the Effectiveness of the 
Utilization of Technically Trained Personnel, U.S. Naval 
Postgraduate School, Monterey, California 1964s 9 
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These results clearly indicate that the Navy should 
give more thought to the problem of using the technical 
abilities of these officers more effectively. Specifically, 
a review of present career planning should be made in 
hopes of arriving at a method of duty assignment planning 
which will allow technically trained officers to be assigned 
associated billets compatible with their education at the 
earliest possible date after completion of this education. 
This of course must be done in light of providing the 
officer with the operational experience necessary for 
Military Command.^ The authors are certain that the 
Navy is aware of this problem, but may not be cognizant 
of the fact that the use of officers not trained in such 
specialties as Operations Analysis results in a large loss 
of effectiveness that could be realized by more judicious 
use of the available corps of technically trained officers. 
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SECNAV Instruction 1520.4 dated 7 March 1963s 



2 
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Ill c Prediction of performance in Operations Analysis 
Training. 

During the gathering and analysis of the data for the 
previous parts of this paper the authors discovered what 
appeared to be another fruitful avenue of investigation; 
the prediction of performance in the Operations Analysis 
Curriculum. Since Q»P«R. is accepted universally as a 
measure of a person's knowledge of the technical tools 
acquired during his training, it would be especially welcome 
if some means of predicting, before training commenced, the 
approximate skill any particular person, or more appropriately, 
any group of persons would aquire. If such a predictor 
could be developed, any input group to the Operations 
Analysis Curriculum could be selected so as to yield maximum 
benefit to the Navy for the time and money expended on the 
training. 

In an attempt to develop some prediction relationships, 
a number of statistics were considered as variables. Using 
QoP.R. as the variable to be predicted nine other statistics 
were treated as the predictors; 

1. Score on Part I of the test vehicle. 

2. Verbal score on the Graduate Record Examination. 

3. Quantitative score on the Graduate Record Examination. 
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i+. Advanced score on the Graduate Record Examination. 

5. Time since graduation or time in an Operations 
Analysis billet. 

6. Score Part II of the test vehicle. 

7. Score Part III of the test vehicle. 

8. Category as defined previously for responders. 

9. Sum of Parts I, II 9 and III. 

A standard linear regression analysis (see Appendix IV) 
■was performed on a CDC 1604 to obtain the prediction 
equations. The level of significance on the F-test used 
was fixed at 0.01 for all runs. As previously mentioned 
in Part II of this paper one run was made using the scores 
of Part II and then Part III of the test vehicle as the 
dependent variable. Many other equations were also obtained^ 
not with the purpose of predicting Q.P.R. specifically but 
for obtaining as much information about the correlations of 
the various statistics as possible. A complete summary of 
these equations is contained in Appendix IV. Initially three 
different groups of data points were useds 

I . All Operations Analysis students and graduates 
for which Q.P.R,, and the G.R.E, scores were available. 

II. The class of 1965 . 

III. The class of 1965 and previous Operations Analysis 
graduates whose statistics were available. 
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From each group of data points eight regression 
equations were obtained. All these equations are tabulated 
in Appendix IV « The equations of interest in predicting 
QoP.Ro are necessarily those containing only independent 
variables which are readily available before a person or 
person's start the course of instruction. Of the group 
of variables used in the analysis only Part I and the G.R.E. 
scores would be obtainable prior to enrolement. 

The following is the regression analysis and equations 
obtained in the order of the groups of data points. 

GROUP I 



Var- 



iable 

No. 


Mean 


Standard 

Deviation 


Regression 

Coefficient 


Std. Error 
Reg. Coef. 


2 


50.318 


6.349 


.031 


.011 


3 


579.649 


108.514 


.001 


.001 


4 


673.377 


86.551 


.001 


.001 


5 


308.701 


248.352 


.000 


.000 


1 


1.938 


.625 






Multiple Correlation Coefficient 


.5978 





Computed 
T Value 


Partial 
Corr. Coef. 


Variance 

Added 


Prop. Var. 
Cum . 


2.778 


.311 


7.670 


.259 


I .466 


.170 


1.672 


.056 


1.639 


.189 


.699 


.024 


1 . 446 


.168 


.553 


.019 
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GROUP II 

Var- 



iable 


Standard 


Regression 


Std. Error of 


No. Mean 


Deviation 


Coefficient 


Reg. Coef. 


2 52.228 


5o365 


0 


032 


.013 


3 610 o 833 


76.322 


0 


001 


.001 


k 675*556 


95<>900 


0 


001 


.001 


5 5k5o000 


91.853 


0 


001 


.001 


1 2.043 


.410 








Multiple Correlation Coefficient .6142 






Computed 


Partial 


Variance 


Prop. Var. 


T Value Corr. Coefi 


Added 




Cum. 


2.483 


.407 


1.747 




.296 


-.993 


-.176 


o037 




.006 


. 662 


.118 


.147 




.025 


1.571 


.272 


.292 




.050 







GROUP 


III 




Var- 










iable 




Standard 


Regression 


Std. Error of 


No. 


Mean 


Deviation 


Coefficient 


Reg. Coef. 


2 


52.433 


6.017 


.022 


O 

O 

0 


3 


607.111 


82.754 


-.001 


.001 


4 


682.889 


91.617 


.001 


.001 


5 


554.444 


93.628 


.002 


.001 


1 


2.103 


.413 






Multiple 


5 Correlation Coefficient 


.6118 




Computed 


Partial Variance Prop. Var. 


T Value Corr. Coef. 


Added 


Cum . 


2.289 




.340 1 


.652 


.220 


- .943 




-.148 


.003 


.000 


° 773 




.121 


.292 


.039 


2.712 




.394 


.864 


.115 
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It is readily seen from the above equations that the 
multiple correlation coefficients of the three groups of 
data points are remarkably constant (0.5978> 0.6142, 
0.6118). The authors found this somewhat surprising 
since previous analysis of data had indicated that the 
Q.P.Ro of the 1966 class would not have settled down 
into the same pattern as those of the 1965 class and the 
graduates. It had further been hypothesized that the 
data points obtained from graduates would be somewhat 
distorted due to their high mean Q.P 0 R. and the fact 
that since they were not in school they would have lost 
some of their "test taking ability. " Even though the 
multiple correlation coefficients are fairly constant over 
the three groups of data it is apparent that the class of 
1966 does degrade the results and it is therefore felt that 
either the second or third equation should be used for 
prediction. 

As time passes and more data points are obtained it 
is felt that this method will yield ever better prediction 
equations. The authors leave it to the readers to decide 
if the present equations with approximately 0.34 as the 
standard error of estimate is sufficiently accurate for 
their application. The authors do feel that the results 
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obtained thus far strongly support the hypothesis that the 
predictors (Part I 9 G.R.E. scores) can be used to select 
the makeup of a class or predict the performance of a 
previously selected group. It is further felt that these 
predictors and methodology could be applied to other 
curricula -with equally promising results. 
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CHAPTER III 



CONCLUSIONS AND RECOMMENDATIONS 

Reliability and Validity determinations for the test 
instrument have shown that the basic tool of the proposed 
methodology is a feasible and meaningful model with which to 
measure technical resources. Admittedly there are a number 
of necessarily subjective "proofs" included in this analysis, 
but the authors feel that quantitative determinations have 
served to augment these "proofs," and provide a creditable 
coalescence leading to an overall high degree of adequacy for 
the test vehicle. This logical trajectory had led the authors 
to the conclusion that this test does measure relative 
percentages of technical resources which in turn gives an 
excellent indication of effectiveness in billets requiring these 
resources . 

Analysis of the data garnered from this test model on 
the Operations Analysis Curriculum has objectively indicated 
that: 

(1) The percentage of technical resources retained 
decreases logarithmically (approximately) with time. 

(2) On-the-job training in Operations Analyst billets is 
significantly inferior to formal Operations Analysis training - 
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especially in the area of learning fundamental technical 
knowledge. 

(3) If technical ability is utilized immediately after 
it's acquisition, loss of this resource occurs at a slower 
rate. 

(4) Although Operations Analysis is a field utilizing 
mainly the basic sciences (Mathematics, Physics, Probability, 
etc.), College Graduates (Non Operations Analysis trained) 
holding degrees which require knowledge of most of these 
subjects do not possess nor readily learn the basic tools 

of the Operations Analyst. 

(5) Graduates of the Operations Analysis Curriculum 
(regardless of time since completion of course or use or 
non-use of technical abilities) display three (3) to four (4) 
times the technical knowledge of College Graduates who have 
held Operations Analyst billets for an average of 19 months. 

(6) Use of Part I of the test instrument and the three 
parts of the graduate achievement tests taken by all students 
of USNPGS has yielded a quantitative method of augmenting 
the screening of prospective students to the Operations 
Analysis Curriculum. 

From these conclusions it is recommended that* 

(1) Further investigation of this methodology be under- 
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taken. It is obvious that the sample sizes obtained were 



not large enough to make "bullet proof" statistical conclusions 
but rather serve as indications that the stated postulates 
concerning the use of technical abilities and the effectiveness 
of various groups of officers holding Operations Analyst 
billets are essentially correct. 

(2) The significant disparity in technical abilities of 

the two groups of officers mentioned in 5» (above) definitely 
suggests that Naval Officer assignment policies be reviewed 
in hopes of assigning as many Operations Analysis trained 
officers to Operations Analyst billets as is consistent with 
other requirements. 

(3) Use of the regression equations and correlation 
coefficients presented in Appendix IV would be of consider- 
able use for Operations Analysis Curriculum officials in 
balancing the levels of Operations Analysis Classes in any 
manner consistent with the ever changing policies and needs 
of the Postgraduate School. 

(i+) If future investigations of this type are undertaken 
a more effective method of insuring the test subjects complete 
the required questionaire and test instrument should be invoked. 
The authors were forced to appeal to the examinees sense 
of responsibility, their response being completely voluntary. 
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It is felt that in order to obtain meaningful sample sizes 
from which learning and forgetting curves can be accurately 
constructed, and measures of effectiveness deduced, 
completion of the test vehicle should, in some manner, be 
made mandatory . 
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APPENDIX I 

SCORES ACHIEVED ON TEST INSTRUMENT 
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CATEGORY LA INPUT OPERATIONS ANALYSIS 

CLASS 1966 - PART ONE TAKEN 
DURING FIRST TERM IN 
CURRICULUM 



Examinee PART I 



1 


50.6 


2 


52.6 


3 


35.6 


kr 


45.6 
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54-6 
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50.6 
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53.6 


8 


45.6 
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33.6 
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51 .6 
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60 . 6 
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38.6 
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38.0 


17 


44 • 8 
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43.6 


25 


50.6 


26 
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28 
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29 


52.6 


30 


50.6 


31 


48.6 


32 


49.6 


33 


54.6 


34 


41.6 



Mean = 47*9 
Standard Deviation = 6 . 4 I 
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CATEGORY 2A OPERATIONS ANALYSIS GRADUATES WHO HAVE GONE 

DIRECTLY TO OPERATIONS ANALYST BILLETS 
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PART I PART II PART III 1 ,11, III 

Mean 53«37 23.82 3-96 81.33 

Standard Deviation 7«95 6 . 64 .96 15.8 



CATEGORY 2B OPERATIONS ANALYSIS GRADUATES WHO HAVE NEVER 

HELD BILLETS 
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CATEGORY 3 NON OPERATIONS ANALYSIS GRADUATES HOLDING 
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APPENDIX II 



RELIABILITY 



DATA 
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RELIABILITY CALCULATIONS 



Data from Table #1 
Equations from Chapter II 
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APPENDIX III 



Learning, Relearning 
and Forgetting Curves 
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KEY TO GRAPHS 



Weighted Adjusted Score is the ratio of the Mean Score 
of a particular category to the individual's score on 
Part I multiplied by the individual's score on Part II. 
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TIME, IN MONTHS, SINCE GRADUATION 




CATEGORY 3 



i 









i 



in 

cn v 




r 



i+9 



T 



TIME, IN MONTHS, IN OPERATIONS ANALYST BILLET 



APPENDIX IV 



REGRESSION EQUATIONS 
AND 

CORRELATION COEFFICIENTS 
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GROUP I 



Regression Equations using all Operations Analysis students and 
graduates for which QPR, and the G.R.E. scores were available. 



Variable No . 
1 
2 

3 

4 

5 

6 

7 

8 



Quality Point Rating 

Part I Score 

G.R.E. Verbal Score 

G.R.E. Quantitative Score 

G.R.E. Advanced Score 

Time (in billet or out of school) 

Part II Score 

Part III Score 



CORRELATION COEFFICIENTS 
ROW 1 



1.00000 


.42051 


.39584 


.37374 


.50869 


ROW 2 
.42051 


1.00000 


.47756 


.30214 


.42939 


ROW 3 
.39584 


.47756 


1.00000 


.19532 


.41359 


ROW 4 
.37374 


.30214 


.19532 


1.00000 


.35157 


ROW 5 
.50869 


.42939 


.41359 


.35157 


1.00000 



SAMPLE SIZE 77 

NO. OF VARIABLES 5 NO. OF VARIABLES DELETED 3 (FOR VARIABLES 
DEPENDENT VARIABLE IS NO NO. 1 DELETES, SEE BELOW) 

COEFFICIENT OF DETERMINATION .3574 

MULTIPLE CORR. COEFFICIENT .5978 

SUM OF SQUARES ATTRIBUTABLE TO REGRESSION 1C. 59485 

SUM OF SQUARES OF DEVIATION FROM REGRESSION 19.04868 

VARIANCE OF ESTIMATE .26456 
STD. ERROR OF ESTIMATE .51436 

INTERCEPT (A VALUE) -1.14097 
STD. ERROR OF INTERCEPT .57577 
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ANALYSIS OF VARIANCE FOR THE MULTIPLE LINEAR REGRESSION 
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MULTIPLE CORR. COEFFICIENT .5537 



VARIABLE STD. REG. STD. ERROR COMPUTED PARTIAL VARIANCE PROP. VAR 

NO. MEAN DEVIATION COEFF. OF REG. COE. T VALUE CORR. COE. ADDED CUM. 

2 50.31818 £.34913 .04208 .01049 4.01231 .42507 7.67059 .25876 

7 12.36364 11.87175 .00964 .01008 .95571 .11116 1.09873 .03706 
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MULTIPLE CORR. COEFFICIENT .5397 



VARIABLE STD. REG. STD. ERROR COMPUTED PARTIAL VARIANCE PROP. VAR 

NO. MEAN DEVIATION COEFF. OF REG. COE. T VALUE CORR. COE. ADDED CUM. 

3 579.64935 108.51386 .00137 .00067 2.05674 .23404 5.46559 .18438 

4 673.37662 86.55103 .00184 .00081 2.26467 .25621 1. 66983 .05633 
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VARIABLE STD. REG. STD. ERROR COMPUTED PARTIAL VARIANCE PROP. VAR 
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GROUP II 



Regression Equations using just the class of 1965. 



Variable No. 
1 
2 

4 

5 

6 

7 

8 


Quality Point Rating 

Part I Score 

G.R.E. Verbal Score 

G.R.E. Quantitative Score 

G.R.E. Advanced Score 

Time (in billet out of school) 

Part II Score 

Part III Score 


CORRELATION 


COEFFICIENTS 






ROW 1 
1.00000 


.31319 


.49274 


.43350 


.54445 


ROW 2 
.31319 


1.00000 


.43538 


.26674 


.09552 


ROW 3 
.49274 


.43538 


1.00000 


.44631 


.37425 


ROW 4 
.43350 


.26674 


.44631 


1.00000 


.45348 


ROW 5 

.54445 


.09552 


.37425 


.45348 


1.00000 



SAMPLE SIZE 36 

NO. OF VARIABLES 5 NO. OF VARIABLES DELETED 3 (FOR VARIABLES 
DEPENDENT VARIABLE IS NOW NO. 1 DELETED , SEE BELOW) 

COEFFICIENT OF DETERMINATION .3772 

MULTIPLE CORR. COEFFICIENT .6142 

SUM OF SQUARES ATTRIBUTABLE TO REGRESSION 2.22289 

SUM OF SQUARES OF DEVIATION FROM REGRESSION 3.66991 

VARIANCE OF ESTIMATE .11838 
STD. ERROR OF ESTIMATE .3440? 

INTERCEPT (A VALUE) -.10242 

STD. ERROR OF INTERCEPT .64905 



56 



ANALYSIS OF VARIANCE FOR THE MULTIPLE LINEAR REGRESSION 



02 

-4 

o 

vO 



cn 

© 

a 



B 

M 

E-« 



I 

Pl, 

o 

e 






02 tO 
C"~ CO 
tO to 



U\ i 



OHO 

to oto 

02 O 02 
NvO O 
02 vO tO 

• • • 

02 CO to 



4 H^A 
CO CO 



r\ 4(0 C^- 

4<^ CMA 
■vO v£) 4 O 
0 0 (M<f 
02 O O O 



§ 



0 s 4 CO 02 

O- O- 02 r-l 
vO v£) O- 02 
<fr\ 4 Q 



►4 

< 

M 

<t 

CL 



O UAO CO 
OlAH ^ 
MAtOH 
O O'* H O- 
4HHC\2 
© » • 

I 



^ (AO ^ 

O to 02 to 

CO 02 02 O 
tO OvO O- 
40\0 lA 



o- o 



02 



02 I 



ca^aN 4 

O tO O- O- 

CO o o o 

H O O O 

o o o o 



VMAHvO 

CO tO to H 

02 O O H 
CO O O Q 
O O O O 

• • • • 
I 



O O to 02 02 
OvOQOcO 
lArj O COO 

Kooio^AH 
co co o to -4 

• • • • • 

lAvO lArl 

O- o o 



to COvO Q CO 
t^ro^O co 

C — CO to O CO 
02 CO tO O 4 
02 tO to O O 
• • • • • 
02 O to to 02 
to rH O- 4 

vOvO IA 



02 CO 4 to rH 



vO 



1 



I 

O 



to 



vO 



57 



VARIABLE STD. REG. STD. ERROR COMPUTED PARTIAL VARIANCE PROP. VAR 

NO. MEAN DEVIATION COEFF. OF REG. COE. T. VALUE CORR. COE. ADDED CUM. 

1 2.09978 .41395 .3554^ .66542 .53419 .08748 8.72377 .10111 
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MULTIPLE CORR. COEFFICIENT .6232 



VARIABLE STD. REG. STD. ERROR COMPUTED PARTIAL VARIANCE PROP. VAR 

NO. MEAN DEVIATION COEFF. OF REG. COE. T VALUE CORR. COE. ADDED CUM. 

~2 52.47778 5.98824 .01992 1(50998 1.99589 30443 1.63794 .21725 
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MULTIPLE CORR. COEFFICIENT .6831 



VARIABLE STD. REG. STD. ERROR COMPUTED PARTIAL VARIANCE PROP. VAR 

NO. MEAN DEVIATION COEFF. OF REG. COE. T VALUE CORR.CQE. ADDED CUM. 

1 2.04333 .41032 .31496 .77896 .40433 .06501 6.89233 .08727 
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MULTIPLE CORR. COEFFICIENT .6828 
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MULTIPLE COER. COEFFICIENT .5033 



VARIABLE STD. REG. STD. ERROR COMPUTED PARTIAL VARIANCE PROP. VAR 

NO. MEAN DEVIATION COEFF. OF REG. COE. T VALUE CORR. COE. ADDED CUM. 

~~ 2 52.22778 5.36500 .04164 .01100 3.78482 754445 1.74679 .29^43 

1 2.04333 .41032 
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GROUP III 



Regression Equations using the class of 1965 and previous 
Operations Analysis graduates whose statistics were available. 



Variable No. 
1 
2 

3 

4 

5 

6 

7 

8 



Quality Point Rating 
Part I Score 
G.R.E. Verbal Score 
G.R.E. Quantitative Score 
G.R.E. Advanced Score 
Sum of Parts I, II, and III 
Part II Score 
Part III Score 



CORRELATION COEFFICIENTS 
ROW 1 



1.00000 


.34292 


.41106 


.36267 


.46906 


ROW 2 
.34292 


1.00000 


.47445 


.28329 


.14136 


ROW 3 
.41106 


.47445 


1.00000 


.41974 


.3508 7 


ROW 4 
.36267 


.28329 


.41974 


1.00000 


.51502 


ROW 5 
.46906 


.14136 


.35087 


.51502 


1.00000 



SAMPLE SIZE 45 

NO. OF VARIABLES 5 NO. OF VARIABLES DELETED 3 (FOR VARIABLES 
DEPENDENT VARIABLE IS NOW NO. 1 DELETED, SEE BELOW) 

COEFFICIENT OF DETERMINATION .3743 

MULTIPLE CORR. COEFFICIENT .6118 

SUM OF SQUARES ATTRIBUTABLE TO REGRESSION 2.81137 

SUM OF SQUARES OF DEVIATION FROM REGRESSION 4.69891 

VARIANCE OF ESTIMATE .11747 

STD. ERROR OF ESTIMATE .34274 

INTERCEPT (A VALUE) .02897 

STD. ERROR OF INTERCEPT .53596 
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ANALYSIS OF VARIANCE FOR THE MULTIPLE LINEAR REGRESSION 
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VARIABLE STD. REG. STD. ERROR COMPUTED PARTIAL VARIANCE PROP. VAR 

NO. MEAN DEVIATION COEFF. OF REG. COE. T VALUE CORR. COE. ADDED CUM. 

I 2.10267 .41314 .45906 .6081 5 .75484 02315 7. 55111 .O8583 
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MULTIPLE CORR. COEFFICIENT .6170 
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VARIABLE STD. REG. STD. ERROR COMPUTED PARTIAL VARIANCE PROP. VAR 

NO. MEAN DEVIATION COEFF. OF REG. COE. T VALUE CORR. COE. ADDED CUM. 
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MEAN DEVIATION COEFF. OF REG. COE. T VALUE CORR. COE. ADDED CUM. 

52.43333 £.01744 .03220 .00925 3-48276 .46906 1.65241 .22002 

2.10267 .41314 



GROUP IV 



Regression Equations using only returns from graduates of the 
Operations Analysis Curriculum or officers holding Operations 
Analyst billets. 



Variable No, 
1 
2 

3 

4 

5 



Time (in billet or since graduation) 
Part I Score 
Part II Score 
Part III Score 

Category (24=1.0, SB-2, 2C=3, 23=4* 3=5) 



CORRELATION COEFFICIENTS 
ROW 1 



1.00000 


-.08659 


.20970 


.12543 


ROW 2 
-.08659 


1.00000 


-.18069 


.36119 


ROW 3 
.20970 


-.18069 


1.00000 


-.72901 


ROW 4 
.12543 


.36119 


-.72901 


1.00000 



SAMPLE SIZE 34 

NO OF VARIABLES 4 NO. OF VARIABLES DELETED 1 (FOR VARIABLES 

DEPENDENT VARIABLE IS NOW NO. 3 DELETED* SEE BELOW) 

COEFFICIENT OF DETERMINATION .6740 

MULTIPLE CORR. COEFFICIENT .8210 

SUM OF SQUARES ATTRIBUTABLE TO REGRESSION 2526.80694 

SUM OF SQUARES OF DEVIATION FROM REGRESSION 1222.22490 

VARIANCE OF ESTIMATE 40.74083 

STD. ERROR OF ESTIMATE 6.38285 

INTERCEPT (A VALUE) 5.60460 

STD. ERROR OF INTERCEPT 9.01661 
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MULTIPLE CORR. COEFFICIENT .7368 



VARIABLE STD. REG. STD. ERROR COMPUTED PARTIAL VARIANCE PROP. VAR 

NO. MEAN DEVIATION COEFF. OF REG. COE. T VALUE CORR. COB. ADDED CUM. 

= 1 14 <>^7^47 8.93655 .04358 .02226 1.95769 .34166 4.34186 .04397 

2 52.52235 7.27225 .02446 .02701 .90567 .16585 2.62804 .02662 
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